--- title: MemeFolder keywords: fastai sidebar: home_sidebar summary: "A convenient wrapper around CLIP. Takes a folder of images and a query and returns a ranked list of image filenames." description: "A convenient wrapper around CLIP. Takes a folder of images and a query and returns a ranked list of image filenames." nb_path: "02_folder.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %} {% raw %}

class MemeFolder[source]

MemeFolder(folder_str, clip_model='ViT-B/32', clear_cache=False, use_treemap=True)

Takes an image folder and a CLIP model and calculates the encodings for each image

{% endraw %} {% raw %}
{% endraw %}

The MemeFolder is a class that represents a folder of images, and allows you to search through them by text or by image. The semantic search is powered by OpenAI's CLIP model. The MemeFolder class provides methods for inference and for storing and retrieving embeddings.

For example, here's one made from the small set of memes in the local images folder:

{% raw %} 
small = MemeFolder('images', use_treemap=False)
{% endraw %} {% raw %} 
len(small.names)

92
{% endraw %}

You can predict text from a MemeFolder's methods. For smaller datasets, the simpler method predict_from_text_trees can be used.

{% raw %} 
cats = small.predict_from_text_dict('cat')
cats[:5]

['images/memes/wgglo1jpy4l61.jpg',
 'images/memes/9r4w3vk182l61.jpg',
 'images/memes/750zzwz944l61.jpg',
 'images/memes/5rbzm1yuw6l61.jpg',
 'images/memes/6sya0pp0u2l61.jpg']
{% endraw %}

Let's double check those images using a quick and dirty printi.

{% raw %} 
def printi(images, n = 3, w = 200, start_index = 0):
    for im in images[start_index:start_index + n]:
#         print(f'{im}')
        try:
            display(Image(filename=im, width=w))
        except Exception as e:
            print(e)
# printi(image_names, 1)
{% endraw %} {% raw %} 
printi(cats, 3, 150)
{% endraw %}

Weird! There's only one cat meme in the sample dataset. Let's try with man's other best friend:

{% raw %} 
dogs = small.predict_from_text_dict('dog')
{% endraw %} {% raw %} 
printi(dogs)
{% endraw %} {% raw %} 
assert len(dogs) == len(small)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-126-865d3b4e652c> in <module>
----> 1 assert len(dogs) == len(small)

<ipython-input-102-c2ee00735cda> in __len__(self)
     20     def __repr__(self): return(self.__str__())
     21 
---> 22     def __len__(self): return(len(self.features))
     23 
     24     def preproc_images(self):

AttributeError: 'MemeFolder' object has no attribute 'features'
{% endraw %}

That's better! Now, if your dataset is more than ~1000 images the predict_text_from_trees method will work much faster. Right now the MemeFolder generates a treemap by default. This will slow down the initiation of the folder, but massively speed up inference.

Generating the treemap may be a minor slowdown for smaller datasets. You can prevent it by passing use_treemap = False when creating the MemeFolder. For most purposes the treemap is better though.

{% raw %} 
small = MemeFolder('images')
{% endraw %} {% raw %} 
doggos = small.predict_from_text_trees('dogs')
{% endraw %} {% raw %} 
printi(doggos)
{% endraw %}

The really amazing thing about CLIP is that it does semantic search and text recognition at the same time. So not only can it recognize an image of a dog, it can also see a meme template with the word "dog" above it!

For a more dramatic example, let's search the dataset for "penis" (and hope dearly that there aren't any dick pics in here):

{% raw %} 
penii = small.predict_from_text_trees('penis')
{% endraw %} {% raw %} 
printi(penii)
{% endraw %}

Though printi only shows the top 3 by default, the MemeFolder returns all of the images in ranked order. This can be useful for, e.g., finding the least "penis" images:

{% raw %} 
not_penii = sorted(penii, reverse=True)
{% endraw %} {% raw %} 
printi(not_penii)
{% endraw %}

Enlightening.

Problem

For some reason the treemap method is causing the length of the folder to be shortened. Why is this happening?

{% raw %} 
len(doggos)

62
{% endraw %} {% raw %} 
len(dogs)

92
{% endraw %}